No token padding for train_network #1677
Conversation
|
A.6 states the following:
According to this, token padding is performed during training. I believe A.6 is a report that it is better not to perform padding on CFG. Also |
|
As far as I can tell, the Illustrious report is saying the padding tokens unintentionally learned information about the images, which causes adverse effects when they're included during inference especially since most UIs will optimize CFG by inserting padding so cond and uncond can be batched together. As mentioned above, everything must be padded to the same length for batching to work, so as a fix they suggest masking out the padding tokens during training. |
I hadn't thought of that, but it's certainly possible. However, unlike SD3, there is an option to extend the token length to 150/225 in a bit hacky way. So it seems that the implementation cannot be copied as is... |
When training with Illustrious-xl 0.1, it appears that token padding was disabled during the training process.
https://arxiv.org/pdf/2409.19946
Illustrious: an Open Advanced Illustration Model A.6
Testing with token padding enabled seems to result in worse outcomes.
To address this issue, I've added the
--no_token_paddingoption to the training network,allowing users to disable token padding during training for potentially better results.
prompt 1girl, solo
no neg